Review of ranked-based and unranked-based metrics for determining the effectiveness of search engines

Purpose: Traditionally, there have many metrics for evaluating the search engine, nevertheless various researchers’ proposed new metrics in recent years. Aware of this new metrics is essential to conduct research on evaluation of the search engine field. So, the purpose of this study was to provide an analysis of important and new metrics for evaluating the search engines. Methodology: This is a review article that critically has investigated the effectiveness Metrics of evaluation of search engines. For this purpose, the words “evaluation metrics”, “evaluation measure” “search engine evaluation”, “information retrieval system evaluation”, “relevance evaluation measure” and “relevance evaluation metrics” were searched in “MagIran” and “Sid” databases and Google Scholar search engine. In this study, we were gathered the articles. Then, we were used existing approaches in evaluation of information retrieval systems to analyze evaluation metrics of search engines. In other words, the descriptive-analytical approach was used to review the search engine assessment metrics. Findings: The theoretical and philosophical foundations determine the type of research methods and techniques. There are two well-known “system-oriented” and “user-oriented” approaches to evaluating information retrieval systems. So, researchers such as Sirotkin (2013) and Bama, Ahmed, & Saravanan (2015) group the precision and recall metrics in a system-oriented approach. They also believe that Average Distance, normalized discounted cumulative gain, Rank Eff and b pref are rooted in the user-oriented approach. Nowkarizi and Zeynali Tazehkandi (2019) introduced comprehensiveness metric instead of Recall metric. They argue that their metric is rooted in a user-oriented approach, while this goal is not fully met. On the other hand, Hjørland (2010) emphasizes that we need a third approach to eliminate this dichotomy. In this regard, researchers such as Borlund & Ingwersen (1998), Borlund (2003), and Thornley and Gibb (2007) have also mentioned a third approach to evaluating information retrieval systems that refer to interact and compose two mentioned approaches. In this regard, Borlund and Ingwersen (1998) proposed a Jaccard Association and Cosine Association measures to evaluate information retrieval systems. It seems that these two metrics have failed to compose the system-oriented and user-oriented approaches completely. However, these two metrics need further investigation. Conclusion: Search engines consist of different components including, Crawler, Indexer, Query Processor, Retrieval Software, and Ranker. Users want to use the most efficient search engines for retrieving required information resource. They are not interested in the technical issues and how each of the search engine component works. Therefore, to suggest the most efficient search engine for users, the performance of all the search engine components must be measured. Each of the effectiveness metrics, measures a specific component of search engines. To measure all components of them, it is suggested to researchers of the evaluation of search engine to select metrics from all three mentioned groups in their research.